LABORATORY  FOR  INFORMATION  AND  DECISION  SYSTEMS 
Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts  02139 


Final  Technical  Report  for 
Grant  FA9550-04- 1-0351 


GRAPHICAL,  GEOMETRIC,  AND  LEARNING/OPTIMIZATION- 
BASED  METHODS  IN  STATISTICAL  SIGNAL  AND  IMAGE 
PROCESSING,  OBJECT  RECOGNITION,  AND  DATA  FUSION 


PRINCIPAL  INVESTIGATOR:  PROF.  ALAN  S.  WILLSKY 


Submitted  to:  Dr.  Jon  Sjogren 
AFOSR/NE 
Physics  and  Electronics 
875  North  Randolph  Street 
Suite  325,  Room  3112 
Arlington,  VA  22203 


March  1,  2008 

20080502069 


3 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  NO.  0704-0188 


Public  Reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 

gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comment  regarding  this  burden  estimates  or  any  other  aspect  of  this  collection 
of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  information  Operations  and  Reports,  1215  Jefferson  Davis  Highway, 

Suite  1204  Arlineton  VA  22202-4302.  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188,)  Washington,  DC  20503. 

1.  AGENCY  USE  ONLY  (  Leave  Blank) 

2.  REPORT  DATE  March  1,2008 

3.  REPORT  TYPE  AND  DATES  COVERED 

Final  Report,  5/1/04  -  2/29/08 

4.  TITLE  AND  SUBTITLE 

Graphical,  Geometric,  and  Leaming/Optimization-Based  Methods  in  Statistical  Signal  and  Image 
Processing,  Object  Recognition,  and  Data  Fusion 

5.  FUNDING  NUMBERS 

FA9550-04- 1-0351 

6.  AUTHOR(S)  Alan  S.  Willsky 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Laboratory  for  Information  and  Decision  Systems 

Massachusetts  Institute  of  Technology 

Cambridge,  MA  02139 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Office  of  Scientific  Research 

875  North  Randolph  Street 

Arlington,  VA  22203 

10.  SPONSORING  /  MONITORING 

AGENCY  REPORT  NUMBER 

AFRL-SR-AR-TR-08-0227 

11.  SUPPLEMENTARY  NOTES 

The  views,  opinions  and/or  findings  contained  in  this  report  are  those  of  the  author(s)  and  should  not  be  construed  as  an  official 
Department  of  the  Army  position,  policy  or  decision,  unless  so  designated  by  other  documentation. 

12  a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited. 

12  b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

This  report  summarizes  our  accomplishments  under  this  grant.  Our  research  covers  several  interrelated  areas:  (a) 
the  use  of  graphical,  hierarchical,  and  multiresolution  representations  for  the  development  of  statistical  modeling 
methodologies  for  complex  phenomena  and  for  the  construction  of  scalable  algorithms  for  the  fusion  of  multiple 
heterogeneous  sources  of  information;  (b)  the  development  of  first  principles  methods  for  constructing  statistical  models 
for  shapes  and  for  the  use  of  these  models  in  developing  robust  and  statistically  optimal  methods  of  shape  estimation  and 
recognition;  and  (c)  the  development  of  new  statistical  learning  and  optimization  algorithms  for  feature  extraction,  signal 
and  image  restoration,  and  sensor  fusion.  Our  research  blends  methods  from  several  fields — statistics  and  probability, 
signal  and  image  processing,  mathematical  physics,  scientific  computing,  statistical  learning  theory,  and  differential 
geometry — to  produce  new  approaches  to  emerging  and  challenging  problems  in  signal  and  image  processing,  and  each 
aspect  of  our  program  contains  both  fundamental  research  in  mathematical  sciences  and  important  applications  of  direct 
relevance  to  Air  Force  missions.  In  particular,  our  research  is  relevant  to  automatic  target  recognition  based  on  synthetic 
aperture  radar  and  laser  radar  imagery;  wide-area  surveillance  and  information  preparation  of  the  battlefield;  global 
awareness  and  higher-level  fusion  for  situational  assessment;  and  fusion  of  multiple  heterogeneous  sensors.  In  all  of  these 
areas  we  have  contacts  and  interactions  with  AFRL  staff  and  with  industry  involved  in  Air  Force  programs. 

14.  SUBJECT  TERMS 

Statistical  signal  and  image  processing;  multiresolution  methods,  sensor  and  information  fusion; 
shape  modeling,  estimation,  and  recognition;  ATR;  situational  awareness 


27 


$ 


) 


) 


16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 

OR  REPORT 

UNCLASSIFIED 

18.  SECURITY  CLASSIFICATION 

ON  THIS  PAGE 

UNCLASSIFIED 

19.  SECURITY  CLASSIFICATION 

OF  ABSTRACT 

UNCLASSIFIED 

20.  LIMITATION  OF  ABSTRACT 

UL 

2 


I.  Summary:  Objectives  and  Status  of  Effort 

In  this  report  we  summarize  our  accomplishments  under  the  research  program 
supported  by  Grant  FA9550-04- 1-0351.  Our  research  covers  several  interrelated  areas: 

(a)  the  use  of  graphical,  hierarchical,  and  multiresolution  representations  for  the 
development  of  statistical  modeling  methodologies  for  complex  phenomena  and  for  the 
construction  of  scalable  algorithms  for  the  fusion  of  multiple  heterogeneous  sources  of 
information;  (b)  the  development  of  first  principles  methods  for  constructing  statistical 
models  for  shapes  and  for  the  use  of  these  models  in  developing  robust  and  statistically 
optimal  methods  of  shape  estimation  and  recognition;  and  (c)  the  development  of  new 
statistical  learning  and  optimization  algorithms  for  feature  extraction,  signal  and  image 
restoration,  and  sensor  fusion.  Our  research  blends  methods  from  several  fields — 
statistics  and  probability,  signal  and  image  processing,  mathematical  physics,  scientific 
computing,  statistical  learning  theory,  and  differential  geometry — to  produce  new 
approaches  to  emerging  and  challenging  problems  in  signal  and  image  processing,  and 
each  aspect  of  our  program  contains  both  fundamental  research  in  mathematical  sciences 
and  important  applications  of  direct  relevance  to  Air  Force  missions.  In  particular,  our 
research  is  relevant  to  automatic  target  recognition  based  on  synthetic  aperture  radar  and 
laser  radar  imagery;  wide-area  surveillance  and  information  preparation  of  the  battlefield; 
global  awareness  and  higher-level  fusion  for  situational  assessment;  and  fusion  of 
multiple  heterogeneous  sensors.  In  all  of  these  areas  we  have  contacts  and  interactions 
with  AFRL  staff  and  with  industry  involved  in  Air  Force  programs. 

The  principal  investigator  for  this  effort  is  Professor  Alan  S.  Willsky.  Prof. 
Willsky  is  assisted  in  the  conduct  of  this  research  by  Dt^John  Fisher,  principal  research 
scientist  in  Prof.  Willsky's  group,  by  Dr.  Mujdat  Cetin,  research  scientist  in  Prof. 
Willsky’s  group,  by  Dr.  Sujay  Sanghavi,  a  post-doctoral  researcher  in  Prof.  Willsky’s 
group,  and  by  several  graduate  research  assistants  as  well  as  additional  thesis  students  not 
requiring  stipend  or  tuition  support  from  this  grant.  In  the  next  section  we  briefly 
describe  our  recent  research  efforts;  in  Section  III  we  indicate  the  individuals  involved  in 
this  effort;  in  Section  IV  we  list  the  publications  supported  by  this  effort;  and  in  Section 
V  we  discuss  several  other  topics  including  honors  received  by  researchers  involved  in 
this  project,  transitions,  and  plans  for  future  transitions. 


4 


II.  Accomplishments 

In  this  section  we  briefly  describe  our  research  supported  under  this  grant.  We 
limit  ourselves  here  to  a  succinct  summary  and  refer  to  the  publications  listed  at  the  end 
of  this  report  for  detailed  developments.  However,  we  do  note  here  that  our  work 
continues  to  have  significant  impact,  both  in  terms  of  DoD-related  activities  and 
transitions  in  progress  (Section  V)  and  in  terms  of  recognition  from  the  research 
community. 

2.1  Graphical,  Multiresolution,  and  Hierarchical  Modeling  and  Fusion 

The  research  described  in  this  section  is  developed  in  great  detail  in  a  number  of 
papers  and  reports  [2,  6-7,  9-10,  12-14,  17-19,  23-24,  29-33,  35,  38, 43, 48-49,  55-63,  70- 
76,  83-89,  93-105].  The  overall  objective  of  this  portion  of  our  research  is  the 
development  of  methods  for  constructing  stochastic  models  for  phenomena  that  vary  over 
space,  time,  and  hierarchy  and  that  possess  structure  which  can  be  exploited  to  construct 
efficient  and  scaleable  algorithms  for  statistical  inference. 

a)  During  the  past  year  we  have  continued  our  development  of  so-called  walk-sum 
analysis  for  inference  in  Gaussian  graphical  models.  As  described  in  last  year  s 
report,  walk-sum  analysis  represents  an  expansion  of  the  set  of  information  made 
available  to  a  node  through  successive  message  passing  throughout  a  graphical 
model  (so  that  messages  engage  in  “walks”  throughout  the  network  during  which 
they  are  modified  at  each  node,  so  that  information  is  accumulated  in  the  process). 
Using  this  interpretation,  we  have  a  precise  characterization  of  the  gap  between 
what  Belief  Propagation  computes  for  error  variances  in  Gaussian  models  and 
what  the  exact  computation  should  produce.  This  interpretation  leads  to  the 
tightest  known  sufficient  conditions  for  BP  convergence  as  well  as  to  a  deep 
understanding  of  when  BP  fails.  Moreover,  this  walks-sum  analysis  has  provided 
the  basis  for  the  solution  of  a  long-standing  open  problem,  namely  the 
development  of  easily  checked  conditions  for  the  convergence  of  our  previously 
developed  Embedded  Trees  algorithm.  In  addition,  this  work  also  provides  the 
basis  for  an  adaptive  method  for  choosing  which  updates  should  be  considered  at 
each  stage  in  the  iteration,  where  the  criterion  used  measures  the  incremental 
value-added  of  each  option.  This  idea  has  far  broader  implications,  e.g.,  for 
sensor  resource  management  and  in  other  powerful  algorithms  that  we  have 
developed. 

b)  New  for  this  year  is  another  emerging  class  of  algorithms  based  on  Lagrangian 
relaxation.  In  this  approach  an  overall  graphical  model  is  decomposed  into  a  set  of 
models  each  on  a  tractable  subgraph  of  the  original  graph.  Inference  is  then 
performed  subject  to  the  constraint  that  the  estimates  produced  on  all  of  these 
subgraphs  agree.  Adjoining  these  equality  constraints  via  Lagrange  multipliers 
leads  to  iterative  algorithms  in  which  estimates  are  computed  on  all  graphs 
followed  by  modifying  the  decomposition  to  drive  the  estimates  toward  equality. 
In  addition  to  guarantees  of  convergence  for  estimates,  this  approach  also  yields 
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upper  bounds  on  error  variances  which  can  be  further  tightened  by  optimization  of 
the  weighting  used  in  the  decomposition.  We  view  this  approach  as  the  start  of  a 
very  powerful  and  rich  approach  to  inference  that  we  expect  to  develop  further  in 
the  future. 

c)  We  have  also  continued  our  research  on  what  we  refer  to  as  low-rank  variance 
estimation  methods  for  complex  graphical  models.  The  idea  behind  this  approach 
is  to  construct  low-rank  approximations  to  the  identity  matrix  (!)  with  particular 
properties.  Such  a  representation  leads  directly  to  an  estimate  of  the  variance  at 
every  node  in  the  graph  corrupted  by  “interference”  from  the  cross-correlation 
between  pairs  of  nodes  and  the  dot  product  of  the  corresponding  rows  in  the  low- 
rank  approximation  to  the  identity.  These  leads  to  the  idea  of  choosing  the 
approximation  to  have  orthogonal  rows  when  cross-correlations  are  large  but  not 
worrying  about  their  non-orthogonality  if  the  corresponding  cross-correlation  is 
negligible.  This  leads  to  interesting  graph-coloring  algorithms  for  designing  these 
overcomplete  sets  of  rows,  and,  together  with  randomized  choices  of  signs  on 
these  rows,  we  obtain  unbiased  estimates  of  the  exact  variances  with  guaranteed 
accuracy  for  processes  with  exponentially  decaying  correlations.  For  processes 
with  long-distance  correlations  a  variation  on  this  approach  using  wavelets  -  and 
what  we  refer  to  as  spliced  wavelet  bases  -  yields  equally  powerful  methods  for 
an  even  richer  class  of  processes.  Extension  to  problems  involving  the  fusion  of 
multiresolution  data  is  a  promising  direction  for  the  future. 

d)  We  have  made  significant  process  on  a  line  of  research  that  has  already  yielded 
important  results  and  offers  much  more  for  the  future.  The  focus  of  this  work  is 
on  the  building  of  thinned  and  thus  more  tractable  graphical  models  that 
accurately  approximate  the  statistics  of  more  complex  models.  Specifically,  if  we 
attempt  to  build  graphical  models  with  maximum  entropy  whose  statistics  exactly 
match  those  of  a  specified  graphical  model,  we  will,  in  general  obtain  complex 
models.  However,  if  we  relax  the  constraints— i.e.,  if  we  only  require  that  the 
statistics  of  our  simpler  model  be  close  to  those  of  the  more  complex  one  the 
resulting  max-entropy  model  is  frequently  dramatically  simpler.  We  have  a  first 
paper  on  this  work  that  demonstrates  the  model-thinning  power  of  this  approach 
and  we  are  now  considering  its  extension  to  problems  involving  adding  hidden 
variables  in  ways  in  which  we  can  then  perform  thinning  on  this  expanded  model. 
This  is  of  particular  importance  in  the  context  of  multiresolution  modeling  (see 
the  next  topic). 

e)  We  have  now  completed  a  first  investigation  of  a  new  class  of  Gaussian  models 
on  pyramidal  graphs  -  ones  in  which  each  level  in  the  pyramid  is  itself  a  graphical 
model  but  in  which  there  are  also  inter-scale  interconnections.  Such  models  are 
capable  of  capturing  long-distance  correlations  but  without  the  serious  artifacts 
arising  from  models  defined  on  multiscale  trees.  However,  these  models  are 
defined  on  graphs  with  many  loops,  something  that  has  led  in  the  past  to  very 
complex  algorithms  Thanks  to  the  properties  of  these  models  and  their  other 
properties,  we  have  overcome  these  difficulties  and  produced  a  very  powerful 
suite  of  algorithms.  One  such  algorithm  results  from  another  very  important 
property  of  these  models,  namely  that  the  conditional  correlation  within  any  scale, 
when  conditioned  on  its  coarser-  and  finer-scale  neighbors,  is  dramatically 
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compressed.  This  leads  to  algorithms  reminiscent  of  multipole  algorithms  for  the 
solution  of  PDEs  in  which  coarse-scale  estimates  are  corrected  by  local  in-scale 
computations.  Indeed  using  walk-sum  analysis  as  described  previously,  not  only 
have  we  developed  efficient  algorithms  with  this  structure  that  are  guaranteed  to 
converge  but  we  have  used  the  methods  for  adaptive  choice  of  embedded 
subgraphs  mentioned  previously  to  achieve  very  rapid  convergence.  Moreover, 
these  methods  can  also  be  adapted  to  the  very  important  problem  of  re-estimation, 
i.e.,  rapid  recomputation  of  estimates  to  incorporate  new  data.  Such  problems 
arise  in  complex  mapping  applications  -e.g.,  the  incorporation  of  new  terrain 
elevation  data  into  existing  maps.  In  addition,  these  algorithms  are  perfectly 
suited  to  the  Lagrangian  relaxation  methods  described  previously  and  to  the  very 
accurate  low-rank  variance  estimation  methods,  a  property  that  we  have  exploited 
not  only  for  efficient  estimation  but  also  for  rapid  parameter  estimation. 

f)  A  very  recently-initiated  research  direction  is  that  of  learning  tractable  graphical 
models  from  data,  where  the  criterion  used  is  not  model  accuracy  but  model  utility 
-  in  particular  the  error  exponent  in  discriminating  between  two  high-dimensional 
probability  distributions.  As  one  would  expect,  if  vast  amounts  of  data  are 
available,  the  models  learned  for  the  two  different  probability  distributions  revert 
to  the  best  models  for  each  individually.  However,  when  data  are  limited,  the 
results  can  be  significantly  different.  This  is  of  potentially  great  value  in  many 
contexts  in  which  high-dimensional  data  need  to  be  processed  but  sufficient  data 
are  not  available  to  build  accurate  models  (or  building  such  models  is 
computationally  intractable).  Applications  ranging  from  hyperspectral  data 
analysis  to  multimodal  fusion  for  object  classification  will  benefit  from  this  line 
of  research. 

g)  We  have  also  initiated  new  efforts  on  discrete  optimization  problems  specified  on 
graphical  models,  with  initial  focus  on  the  so-called  maximum  independent  set 
and  matching  problems.  Such  problems  arise  in  a  variety  of  applications 
including  many  involving  resource  management  and  optimization.  Such 
problems  are  naturally  cast  as  integer  programming  problems  which  are  NP-hard. 
Relaxed  versions  of  these  problems  can  be  formulated  in  terms  of  linear 
programs.  Such  a  formulation  can  lead  to  integrality  gaps  and  thus  fail  to  give 
optimal  answers;  however  in  some  cases  the  LP  does  indeed  yield  optimal 
solutions.  Alternatively  these  problems  can  be  formulated  as  MAP  estimation 
problems  on  graphical  models  for  which  the  so-called  max-product  algorithm 
provides  a  general  purpose  algorithm  that  is  only  guaranteed  to  yield  optimal 
answers  for  graphs  without  loops  but  often  works  well  in  other  contexts.  We  have 
made  considerable  progress  in  relating  LP  and  max-product  approaches, 
providing  both  conditions  under  which  either  or  both  are  guaranteed  to  yield 
optimal  estimates  as  well  as  new  algorithms. 

h)  During  this  past  year  we  have  continued  our  work  on  a  very  new  class  of 
graphical  models  motivated  by  problems  in  object  recognition  (see  Sec.  2.2)  in 
which  we  have  considerable  uncertainty  about  the  numbers  of  features  associated 
with  each  part  of  an  object,  the  numbers  of  parts  comprising  an  object,  the 
numbers  of  types  of  objects,  and  the  numbers  of  instances  of  each  object  type  in 
the  scene  being  surveilled.  To  attack  these  complexities  we  have  built  on  the 
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framework  of  so-called  Nonparametric  Bayesian  Methods  exploiting  so-called 
Dirichlet  processes  (and  their  extensions)  which  allow  us,  in  a  simple  manner,  to 
capture  ambiguities  and  uncertainties  at  the  level  of  granularity  of  the  numbers  of 
“things” — features,  parts,  objects,  etc.  We  have  demonstrated  the  power  of  this 
method  in  the  context  of  object  recognition,  and  have  now  initiated  an  effort  to 
exploit  this  framework  for  problems  of  learning  dynamic  models  for  the  motion  of 
non-cooperative  targets. 

i)  One  of  the  important  areas  of  application  and  development  of  efficient  graphical 
inference  algorithms  is  multisensor,  multitarget  data  association,  a  notoriously 
complex  problem.  In  previous  years  we  had  demonstrated  that  our  new  approach 
to  so-called  max-product  and  tree-reweighting  inference  algorithms  could  yield 
remarkably  efficient  solutions  to  optimal  data  association  problems.  In  addition, 
with  an  eye  toward  implementation  in  distributed  sensor  networks,  we  developed 
a  local,  adaptive  version  of  TRP  and  belief  propagation  algorithms  in  which,  at 
each  iteration,  each  node  can  decide  whether  to  transmit  a  message  to  each  of  its 
immediate  neighbors  based  on  whether  the  potential  new  message  differs  in  a 
statistically  significant  manner  from  the  previous  message  that  was  sent  to  that 
neighbor.  Last  year  we  showed  that  this  locally  adaptive  algorithm  can  result  in 
dramatic  reductions  in  computations — and  communications,  if  these  messages 
were  indeed  sent  through  a  sensor  network — with  minimal  decrease  in  association 
performance.  In  recent  work  we  have  extended  these  ideas  to  full  temporal  multi¬ 
object  tracking  and  data  association,  using  ideas  from  Nonparametric  Belief 
Propagation  (discussed  next)  to  devise  novel  and  efficient  algorithms  to  achieve 
scalability  and  overcome  the  combinatorial  explosion  inherent  in  exact  multi¬ 
target  tracking.  We  believe  that  this  opens  the  door  to  extremely  powerful  new 
methods  which  we  are  just  beginning  to  explore,  using  these  new  graphical 
representations  as  well  as  incorporating  the  emerging  methods  mentioned  in  item 
(h)  involving  Dirichlet  processes  to  deal  with  unknown  numbers  of  objects  or 
more  complex  situations  in  which  groups  of  objects  are  moving  as  clusters  (e.g., 
convoys,  formation  flying)  but  we  don’t  know  how  many  clusters  there  are  or 
how  many  (and  which)  objects  are  in  each  cluster. 

j)  As  the  preceding  item  illustrates,  we  have  continued  to  exploit  our  approach  to 
inference  for  graphical  models  that  involve  non-Gaussian  densities — problems  of 
particular  importance  for  various  sensing  modalities  that  provide  measurements  of 
either  bearing  or  range.  These  methods,  which  involve  the  use  of  methods  for 
nonparametric  density  estimation  (for  which  reason  we  refer  to  them  as 
Nonparametric  Belief  Propagation  (NBP)  algorithms),  can  be  viewed  as 
extensions  of  concepts  of  particle  filtering  to  inference  on  graphs — this  extension 
is  highly  nontrivial,  especially  for  graphs  with  loops,  as  the  iterative  computations 
and  generation  of  messages  of  belief  propagation  require  new  ideas  for  generating 
“particles”  to  replace  those  messages.  In  addition  to  developing  the  basic 
methodology,  we  have  demonstrated  its  power  in  several  very  different  contexts 
including  the  target  tracking  application  mentioned  in  the  preceding  paragraph, 
source  localization  in  sensor  networks,  the  computer  vision  application  of  hand¬ 
tracking  in  video  sequences  (see  Section  2.2),  machine  learning  approaches  to 
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object  and  scene  recognition  (see  Section  2.2),  and  tracking  of  dynamically 
evolving  shapes  of  objects  (see  Section  2.2). 

k)  As  we  described  last  year,  motivated  by  issues  of  communications-constrained 
sensor  network  operation  we  have  investigated  two  separate  problems,  with 
considerable  success  in  each.  In  the  first  of  these  we  have  examined  the  impact  of 
errors  in  message  passing  algorithms  (where  the  errors  might  result  from 
“message  censoring”  as  discussed  in  the  context  of  target  tracking  in  paragraph 
(c);  or  they  might  result  from  message  quantization).  By  introducing  a  new 
dynamic  range  measure  of  error,  we  have  been  able  to  develop  both  bounds  and 
stochastic  approximations  on  the  propagation  of  errors  in  belief  propagation 
algorithms.  In  addition,  this  analysis  also  had  yielded  the  best  known  sufficient 
conditions  for  the  convergence  of  belief  propagation  in  loopy  graphs.  In  addition, 
we  have  completed  a  study  of  efficient  computation  of  particle-based  messages, 
combining  information-theoretic  concepts  with  our  NBP  algorithmic  structure.  In 
particular,  since  the  order  in  which  particles  are  transmitted  is  irrelevant,  we  have 
developed  a  very  efficient  multiresolution  representation  of  such  methods  that 
allows  very  efficient  coarse-to-fine  coding  and  transmission  as  well  as  a  very 
effective  framework  for  trading  off  accuracy  of  message  transmission  (i.e.,  the 
level  of  granularity  of  the  transmitted  representations)  with  overall  inference 
quality,  providing  a  tie  directly  between  communication  constraints  on  bit  rates 
and  overall  fusion  accuracy. 

l)  We  have  also  made  significant  progress  in  an  investigation  that  brings  together 
the  field  of  decentralized  team  decision-making  and  message  passing  algorithms 
on  graphs.  In  particular,  for  the  case  of  a  directed  set  of  sensing,  decision,  and 
communication  nodes  (so  that  each  node  receive^  its  own  measurements  together 
with  bits  from  its  “parent”  nodes  and  then  makes  decisions  resulting  in  bits 
transmitted  to  its  “children”)  we  have  shown  that  so-called  person-by-person  team 
optimization  can  be  achieved  via  a  message  passing  algorithm.  This  emphasizes 
that  in  communication-limited  contexts  with  distributed  agents,  the  agents  must 
organize  themselves  and,  in  particular,  design  communication  protocols  for  the 
generation  and  interpretation  of  messages  within  the  agent  network.  We  have 
now  written  an  extensive  paper  on  this  work  and  demonstrated  its  value  in 
designing  decision  networks  that  may  differ  in  structure  from  that  of  the 
underlying  variables  being  estimated.  Moreover  we  have  begun  to  develop  an 
undirected  version  of  this  framework  -  a  nontrivial  extension  as  such  a  framework 
in  principle  allows  feedback  so  that  making  a  decision  on  what  to  communicate 
must  also  be  based  on  the  impact  that  that  communication  will  have  on  what  will 
be  communicated  back  to  the  transmitting  node. 

m)  We  have  also  developed  a  new,  first  principles  probabilistic  approach  to  Markov 
modeling  on  trees,  together  with  a  start  on  the  nontrivial  generalization  to  graphs 
with  loops.  Interestingly  this  approach  identifies  reduced  sets  of  conditional 
independence  relationships  that  need  to  be  verified  either  in  determining  if  a 
particular  set  of  variables  are  Markov  or  in  designing  hidden  variable 
representations  to  ensure  Markovianity.  The  former  interpretation  of  our  results  is 
of  great  importance  in  the  context  of  the  estimation  of  the  structure  among  a  set 
of  observed  variables — e.g.,  to  identify  statistical  links  among  them  as  well  as 
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conditional  independencies,  a  topic  sometimes  referred  to  as  link  discovery.  The 
latter  interpretation  is  of  importance  in  building  graphical  models  with  tractable 
structure,  possibly  by  incorporating  hidden  variables,  in  order  to  adequately 
approximate  the  statistical  relationships  among  a  set  of  variables  of  interest. 


2.2  Geometric  Modeling,  Shape  Estimation,  and  Object  Extraction  and  Recognition 

The  research  described  in  this  section  deals  with  efficient  algorithms  for  large- 
scale  optimal  estimation  and  is  reported  in  detail  in  [3,5,1 1,15-16,22,26,  36-37,  39,  41, 
46-47,  58-59,  70-71,  92].  The  general  objective  of  this  part  of  our  research  is  the 
development  of  statistically  robust  methods  for  segmentation,  shape  estimation,  and 
object  recognition.  Much  of  our  work  in  this  area  has  focused  on  so-called  curve 
evolution  methods  and,  in  particular,  on  developing  statistically-based  curve  evolution 
algorithms.  However,  we  now  also  have  some  research  that  exploits  ideas  from  graphical 
models  described  in  the  preceding  subsection: 

a)  The  use  of  particle-based  methods  as  in  NBP  leads  naturally  to  the  question  of 
using  Monte  Carlo  methods  to  sample  from  curve/shape  distributions 
directly — i.e.,  to  generate  “particles”  that  correspond  to  complete  curves.  We 
have  now  developed  a  methodology  for  doing  this  -  a  nontrivial  development 
as  the  use  of  Metropolis-Hastings  algorithms  required  developing  so-called 
detailed  balance  acceptance  rules  that  are  needed  to  guarantee  that  samples 
are  generated  by  the  desired  shape  distribution.  We  have  also  developed 
methods  for  displaying  the  uncertainty  in  the  resulting  extracted  shapes  -  a 
feature  that  we  believe  will  be  of  great  importance  in  object  recognition 
applications.  One  of  the  appealing  aspects  of  this  sampling  framework  is  that, 
with  the  detailed  balance  issue  now  solved,  it  is  relatively  easy  to  include 
features  in  the  distribution  that  are  easily  used  for  acceptance-rejection  of 
samples  but  are  not  easily  incorporated  into  curve  evolution  methods.  Several 
papers  are  in  progress. 

b)  One  of  the  major  areas  of  our  current  and  future  research  in  this  area  is  that  of 
incorporating  prior  information  about  shape  into  curve  evolutions.  This  is 
particularly  important  for  problems  in  which  image  SNR  is  low  or  in  which 
the  objects  of  interest  are  partially  occluded.  Major  issues  here  include  the 
development  of  methods  for  constructing  prior  probability  distributions  on 
shapes  from  examples  and  the  incorporation  of  these  priors  into  curve 
evolution  formalisms.  Our  initial  work  in  this  area  used  a  set  of  training 
examples  to  construct  a  set  of  “eigenshapes,”  which  then  are  used  to  provide  a 
linear  parameterization  of  a  set  of  shapes,  where  the  parameters  of  that  linear 
parameterization  is  then  estimated  as  part  of  the  curve  evolution  process. 
Results  on  both  military  and  medical  images  in  both  2-D  and  3-D  have 
demonstrated  that  this  methodology  has  a  great  deal  of  promise.  In  addition, 
we  have  been  working  to  move  beyond  these  linearly-parameterized  methods 
in  several  different  directions.  The  first  of  these  methods  involves  postulating 
that  the  model  to  be  learned  from  training  examples  is  a  mixture  of  two  or 
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distributions  each  of  which  is  well  characterized  by  principal  component 
analysis.  This  introduces  a  hidden  variable  for  each  training  sample— i.e.,  the 
component  of  the  mixture  to  which  it  corresponds — which  in  turn  leads  to  a 
new  EM-based  algorithm.  Results  demonstrate  the  power  of  this  extension  to 
classify  shapes  and  model  their  variability.  A  second  approach  we  are  taking 
is  that  of  learning  nonparametric  models  for  shapes  given  a  set  of  training 
samples.  Nonparametric  density  estimation  methods  require  the  use  of  a 
distance  metric  between  pairs  of  shapes,  and  our  work  has  led  us  to  use  two 
natural  metrics,  each  of  which  leads  to  a  different  curve  evolution.  Both  of 
these  have  been  shown  to  have  considerable  promise  for  recognizing  and 
segmenting  shapes  that  can  have  considerable  variability  or  be  subject  to 
partial  occlusion.  We  are  also  developing  new  methods  that  can  incorporate 
human  or  expert  input  -  e.g.,  in  the  form  of  partial  segmentations  -  to  help 
guide  both  curve  evolution  as  well  as  Monte  Carlo  sampling. 

c)  We  have  also  made  major  progress  on  the  development  of  methods  for  space- 
time  tracking  of  curves  or  boundaries  that  evolve  in  time  themselves.  Our 
approach  to  this  problem  involves  developing  and  exploiting  models  for  the 
probabilistic  evolution  of  such  curves — in  essence  developing  temporal 
Markov  models  for  these  curves.  Our  approach  combines  (i)  the  idea  of  using 
low-dimensional  parameterizations  of  curves  (e.g.,  using  principal 
components);  (ii)  information-theoretic  methods  to  learn  nonparametric 
statistical  models  for  the  dynamics  of  those  parameterizations;  (iii)  non¬ 
parametric  belief  propagation  to  exploit  these  dynamic  models  for  the  tracking 
of  curves;  and  (iv)  curve  evolution  methods  based  on  level  sets.  Very 
promising  results  have  been  obtained  in  a  soon-to-be-completed  Ph.D.  thesis. 

d)  Our  work  on  NBP  has  found  its  way  into  a  line  of  inquiry  not  involving  curve 
evolution  methods,  namely  direct  tracking  of  objects  that  have  constrained 
motion  (such  as  articulation  of  limbs,  etc.),  using  graphical  models  to  capture 
these  constraints.  The  surrogate  application  used  in  this  study  is  that  of  hand¬ 
tracking,  and  we  have  had  good  success  in  developing  methods  that  are 
superior  to  those  previously  developed. 

e)  Finally,  the  work  described  in  the  preceding  section  on  Dirichlet  processes  has 
provided  the  basis  for  machine  learning  methods  to  learn  graphical  models  for 
scenes,  objects  and  parts— i.e.,  to  learn  relationships  among  these  with  very 
limited  a  priori  information.  This  is  a  very  powerful  approach  to  building 
complex  models  and  one  that  we  believe  is  just  the  tip  of  the  iceberg  e.g., 
our  current  approach  uses  fixed  feature  extraction  methods  for  video  imagery 
and  then  builds  models  based  on  these  features.  More  sophisticated 
approaches — and  ones  that  tie  in  well  with  our  work  on  wide-area  SAR 
described  in  the  next  section — would  involve  feedback  in  that  the  front-end 
feature  extraction  required  might  depend  on  the  likely  hypotheses  for  the 
objects  in  the  scene  being  imaged. 


2.3  Machine  Learning  and  Optimization  Methods  for  Signal  and  Image  Processing, 
Fusion,  and  Feature  Extraction 
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The  research  described  in  this  section  deals  with  methods  for  complex  signal, 
image,  and  data  analysis  using  methods  of  machine  learning  and  optimization-  based 
formulations.  Our  research  is  described  in  [1-2, 4,  8,  14,  21,  27-28,  34,  30, 42, 44-45,  50- 
54,  66-69,  77-83,  90-92,  105-107],  Our  research  has  led  to  the  following  lines  of  inquiry 
and  results: 

a)  An  area  in  which  we  have  made  considerable  progress  this  year  is  that  of 
blending  nonparametric  estimation  methods  (such  as  particle  filtering) 
together  with  learning-based  optimization  algorithms  (e.g.,  approximate 
dynamic  programming)  to  examine  challenging  and  important  problems  in 
sensor  resource  management.  We  have  had  considerable  success  in 
developing  methods  in  several  directions  including:  (i)  switching  among 
multiple  sensing  modes — a  problem  of  great  importance  in  exploiting  multi¬ 
modal  radar;  (ii)  switching  field  of  regard  for  a  sensor  (i.e.,  deciding  when  to 
look  where);  (iii)  dealing  with  limited  communications  in  designing  near- 
optimal  systems  for  tracking  and  track  hand-off  in  sensor  networks;  and,  most 
recently,  (iv)  developing  bounds  on  the  performance  loss  in  using  tractable 
suboptimal  strategies  as  opposed  to  truly  optimal  (but  not  computable)  ones. 

A  number  of  conference  and  journal  papers  have  been  written  or  are  in 
progress. 

b)  Our  work  on  learning  object  dynamics  using  Hierarchical  Dirichlet  processes 
represents  a  new  thrust  for  our  research  on  learning  models  for  complex 
dynamic  phenomena.  We  have  also  continued  our  work  on  exploiting  ideas 
from  nonparametric  statistics,  information  theory,  and  machine  learning  for 
the  construction  of  dynamic  models  from  complex  signals  in  an  unsupervised 
learning  context.  The  principle  we  have  adopted  in  this  and  in  our  other  work 
in  this  area  is  that  of  maximizing  mutual  information.  In  particular,  in  this 
context,  the  objective  is  to  identify  functionals  of  the  past  of  a  signal  that  have 
maximal  mutual  information  with  the  next  value  of  the  signal.  In  the  process 
of  performing  that  optimization  using  nonparametric  statistical  methods,  we 
also  build  a  model  for  the  transition  probability  for  the  process,  i.e.,  the 
conditional  pdf  for  the  next  value  of  the  signal  given  these  maximally 
informative  functionals  of  the  past.  This  pdf,  then,  serves  as  a  dynamic  model 
of  the  phenomenon  which  can  be  used  for  signal  prediction,  simulation, 
discrimination,  and  estimation.  As  discussed  in  Section  2.2,  this  method  is 
now  finding  application  in  our  work  on  tracking  dynamically  evolving  shapes. 
In  addition,  we  are  using  these  methods  in  order  to  detect  and  identify 
statistical  links  among  multiple  data  streams  (e.g.,  audio  and  video).  These 
methods  relate  closely  to  the  ideas  mentioned  in  Section  2.1,  item  (i)  for 
determining  the  statistical  links  among  multiple  variables. 

c)  A  continuing  and  very  active  component  of  our  research  focuses  on 
variational  methods  to  produce  enhanced  images  and  reconstructions  for  SAR, 
ISAR,  and  more  general  array  processing  applications.  In  particular,  by 
putting  particular  penalties  (e.g.,  Lp,  with  p  <  1)  either  on  the  reconstructed 
image  or  on  the  gradient  of  the  reconstructed  image,  we  have  shown  that  we 
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can  produce  remarkably  sharp  images  of  point  scatterers  or  regions  and  can 
also  correct  for  phase  errors  due  to  target  motion — an  extremely  important 
problem  in  SAR  imaging  of  moving  targets  or  to  other  sources  (including 
timing  errors  to  array  element  location  errors).  Moreover,  in  contrast  to  many 
other  superresolution  methods  (e.g.,  MUSIC,  Capon’s  method),  our  method 
can  resolve  multiple  scattering  effects  that  are  highly  correlated — e.g.,  due  to 
the  presence  of  multipath  effects.  In  one  part  of  our  recent  research  we  have 
developed  new  variational  approaches  for  array  processing  that  work  well  for 
broadband  sources  and,  in  particular,  for  sources  that  generate  multiple 
harmonics  (e.g.,  as  are  present  in  any  motor  or  machinery).  In  other  parts  we 
have  developed  new  methods  for  coherent  imaging  for  wide-aperture  and 
wide-but-sparse  aperture  SAR  that  have  attracted  considerable  interest  from 
AFRL/SN. 

d)  In  this  last  part  of  our  research  we  have  taken  a  deeper  look  at  marrying  SAR 
physics  with  nonparametric  statistical  learning  methods  for  constructing 
probabilistic  models  for  multiresolution  imagery.  In  particular  consider  the 
formation  of  SAR  imagery  based  on  a  given  full  aperture  of  data.  If  we  use 
the  entire  aperture,  we  obtain  imagery  at  the  finest  resolution  resolvable  using 
that  data.  However,  to  do  this  we  in  essence  must  assume  that  all  scattering  is 
isotropic,  i.e.,  that  the  response  from  significant  scatterers  is  constant  across 
the  entire  aperture.  For  many  important  scattering  mechanisms  this  is  not  the 
case  at  all,  and  this  anisotropy  is  critical  to  distinguishing  one  scatterer  type 
from  another.  Suppose  then,  that  in  addition  to  forming  an  image  using  the 
entire  aperture,  we  also  form  three  images  each  using  half  of  the  aperture:  one 
image  using  the  right  half,  one  the  left,  and  one  using  a  centered  half-aperture. 
If  indeed  there  are  anisotropic  scatterers,  we  might  expect  that  there  would  be 
differences  in  the  responses  in  each  of  these  half-apertures  and  hence  in  the 
images  formed  using  them  (note  that  these  images  would  have  pixel  sizes 
twice  as  large  as  the  ones  in  the  finest  scale  imagery).  Iterating  this  process, 
we  can  imagine  forming  a  vector  of  images  at  each  of  a  sequence  of  scales 
corresponding  to  progressively  smaller  subapertures.  By  looking  across  scale, 
then,  we  would  expect  not  only  to  find  statistical  variability  due  to  speckle  but 
also  any  evidence  of  anisotropic  scattering  manifesting  itself  in  statistically 
significant  differences  in  pixel  intensities  in  images  formed  using  different 
subapertures.  During  the  past  year  we  have  initiated  a  new  effort  in  this  area 
that  employs  the  “sparseness  prior”  variational  framework  described  in  the 
preceding  paragraph.  Initial  results  provide  the  basis  for  some  new  “best 
basis”  methods  for  imaging  that  avoid  exhaustive  search  of  subapertures 
through  a  modified  coarse-to-fine  search  with  intelligent  back-tracking.  We 
believe  that  there  is  much  more  that  can  be  done  in  this  area.  For  example, 
one  very  promising  direction  for  future  work  is  that  of  coupling  these  front- 
end  algorithms  with  back-end  object  recognition  using  the  framework  of 
Dirichlet  processes  for  object  recognition  described  in  the  preceding  section. 
In  particular,  we  expect  that  by  building  object  models  that  couple  object 
models  with  anisotropy  properties  we  will  be  able  to  develop  algorithms  in 
which  object-level  hypotheses  will  drive  front-end  signal  processing.  This 
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offers  the  possibility  of  a  significant  conceptual  and  algorithmic  leap  over 
current  methods  (e.g.,  the  current  form  of  the  so-called  “PEMS  Loop”  in  the 
algorithms  developed  under  the  MSTAR  program). 
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V.  INTERACTIONS/TRANSITIONS 


In  this  section  we  summarize  our  recent  interactions  and  plans  for  transitions 
associated  with  research  supported  by  AFOSR  Grant  FA9550-04- 1-0351,  as  well  as 
listing  some  important  honors  received  by  members  of  our  research  team. 

Honors 

(1)  Dr.  Alexander  Ihler  received  two  separate  Outstanding  Student  Paper  Awards, 
one  for  his  work  on  localization  in  sensor  networks  [3 1  ]  and  one  for  his  work  on 
message  errors  in  belief  propagation  [33]. 

(2)  Prof.  Alan  Willsky  received  an  Honorary  Doctorate  from  Universite  de  Rennes 
(France)  in  October  2005.  Prof.  Willsky  is  one  of  two  individuals  receiving  this 
degree  as  part  of  the  celebration  of  the  25th  anniversary  of  the  founding  of  IRIS  A 
(Institut  de  Recherche  en  Informatique  et  Systemes  Aleatoires).  A  two-day 
workshop  in  honor  of  the  recipients  will  be  held. 

(3)  Mr.  Lei  Chen  received  the  Outstanding  Student  Paper  Award  at  the  Fusion  2005 
Conference  in  the  summer  of  2005. 

(4)  Mr.  Jason  Williams  was  also  awarded  the  Outstanding  Student  Paper  Award  at 
Fusion  2005. 

(5)  Mr.  Dmitry  Malioutov  and  Mr.  Jason  Johnson  received  the  Outstanding  Student 
Paper  Award  at  ICASSP  2006. 

Participation/Presentation  at  Meetings 

In  addition  to  the  many  invited  and  contributed  talks  presented  at  various 
meetings  during  the  past  year,  we  also  make  note  of  the  following: 

(1)  Prof.  Willsky  delivered  a  plenary  lecture  at  the  Annual  Review  Meeting  of  the 
Sensing  and  Signals  Program  of  AFOSR,  held  at  NC  State  University  in  May 
2005. 

(2)  In  October  2005  Dr.  Cetin  delivered  an  invited  lecture  at  the  Workshop  on 
Imaging  from  Wave  Propagation  at  the  Institute  for  Mathematics  and  its 
Applications  (IMA)  as  part  of  IMA’s  Thematic  Year  on  Imaging. 

(3)  Prof.  Willsky  was  the  Washington  University  John  Zaborszky  Distinguished 
Lecturer  for  2005-6  and  delivered  a  series  of  lectures  in  February  2006. 

(4)  Prof.  Willsky  gave  an  invited  lecture  in  February  2006  as  part  of  the 
University  of  Michigan’s  Dept,  of  Electrical  Engineering  Distinguished 
Lecture  Series. 

(5)  In  May  2006  Prof.  Willsky  gave  an  invited  lecture  in  Stanford  University’s 
Broad  Area  Colloquium  on  Artificial  Intelligence. 

(6)  In  October  2006  Prof.  Willsky  gave  an  invited  lecture  in  the  seminar  series  at 
the  Dept,  of  Electrical  Engineering,  University  of  Connecticut. 

(7)  In  April  2007  Prof.  Willsky  gave  an  invited  lecture  on  large-scale,  scalable 
data  assimilation  from  remote  sensing  data  at  the  IEEE  -  GEOSS  Workshop, 
Honolulu,  HI. 

Consultative  and  Advisory  Functions 
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We  continue  to  be  actively  engaged  in  a  number  of  activities  relevant  to  the 
research  being  performed  under  our  AFOSR  grant: 

(1)  Prof.  Willsky  has  regularly  acted  as  a  consultant  to  BAE  Systems  Advanced 
Information  Technologies  (BAE- AIT;  formerly  Alphatech,  Inc.)  in  a  number 
of  research  projects  including  ones  that  represent  direct  transitions  of  the 
technology  being  developed  under  our  AFOSR  Grant. 

(2)  Prof.  Willsky  is  serving  on  the  Senior  Review  Panel  for  DARPA’s  POSSE 
(Persistent,  Operational  Surface  Surveillance  and  Engagement)  Program 
which  is  aimed  at  rapid  deployment  of  advanced  ISR  systems  to  active  areas 
of  conflict  (note  that  all  of  the  other  members  of  the  panel  are  either  retired  3- 
and  4-star  generals  or  individuals  who  previously  served  as  Deputy  Assistant 
Secretaries  of  Defense). 

Transitions 

The  following  represent  some  of  the  ongoing  transitions  of  our  work  as  well  as 
some  plans  for  future  transitions: 

(1)  Our  work  on  inference  for  graphical  models — and  especially  our  work  on 
NBP,  and  error  analysis  for  belief  propagation  algorithms — have  been 
transitioned  to  MIT  Lincoln  Laboratory.  There  are  a  number  of  programs  at 
Lincoln  that  are  exploiting  our  research,  but  the  most  concentrated  effort 
involves  transition  to  programs  in  missile  defense  including  target  tracking 
and  discrimination.  The  points  of  contact  at  Lincoln  are  Dr.  Keh-Ping  Dunn 
and  Dr.  David  Choi. 

(2)  Our  work  on  sensor  resource  management  is  also  being  transitioned  to 
Lincoln  Laboratory  where  it  is  also  finding  application  in  missile  defense 
programs.  The  point  of  contact  again  is  Dr.  Dunn. 

(3)  Our  efficient  methodology  for  multiresolution  mapping  and  data  fusion  have 
been  transitioned  to  BAE- AIT  as  part  of  several  programs  on  fusion  of 
multiresolution  and  multipass  data  to  produce  high-fidelity  terrain  maps.  The 
point  of  contact  for  this  work  at  BAE-AIT  is  Dr.  Alan  Chao. 

(4)  Transition  of  our  graphical  estimation  and  optimization  methods  to  BAE-AIT 
for  several  programs.  Most  recently,  Prof.  Willsky  has  been  engaged  with  Dr. 
Mark  Luettgen,  who  heads  AIT’s  Fusion  Technology  and  Systems  Division, 
in  transitioning  these  methods  to  programs  in  distributed  multisensor  fusion 
and  detection,  estimation,  and  tracking  of  terrorist  networks. 

(5)  Dr.  Mujdat  Cetin  has  worked  directly  with  Dr.  Eugene  Lavely  of  BAE-AIT  on 
transitioning  his  sparse  regularization  methods  to  problems  in  radar  signal 
processing  and  image  formation. 
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(6)  Dr.  Mujdat  Cetin’s  methods  for  sparse  regularization  for  radar  signal 
processing  and  SAR  analysis  have  been  transitioned  to  AFRL/SN,  and  Dr. 
Cetin,  in  collaboration  with  Prof.  Randy  Moses  of  Ohio  State  University  have 
been  working  toward  enhancing  this  transition.  In  addition,  Dr.  Cetin  was 
recently  awarded  an  AFRL  grant  under  the  RASER  program  to  transition  his 
methods  to  wide-aperture  SAR  image  formation  and  analysis. 

(7)  Our  work  on  new  graphical  model  methods  for  target  tracking  and  for  object 
recognition  are  being  transitioned  to  Lincoln  Laboratory  through  one  of  our 
students,  Emily  Fox,  who  is  working  with  Dr.  Keh-Ping  Dunn  and  Dr.  David 
Choi  of  Lincoln. 
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